<?php
/**
 * <https://y.st./>
 * Copyright © 2019 Alex Yst <mailto:copyright@y.st>
 * 
 * This program is free software: you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation, either version 3 of the License, or
 * (at your option) any later version.
 * 
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
 * GNU General Public License for more details.
 * 
 * You should have received a copy of the GNU General Public License
 * along with this program. If not, see <https://www.gnu.org./licenses/>.
**/

$xhtml = array(
	'<{title}>' => 'Components of data mining',
	'<{subtitle}>' => 'Written in <span title="Data Mining and Machine Learning">CS 4407</span> by <a href="https://y.st./">Alexand(er|ra) Yst</a>, finalised on 2019-02-13',
	'<{copyright year}>' => '2019',
	'takedown' => '2017-11-01',
	'<{body}>' => <<<END
<div class="APA_title_page">
	<p>
		Components of data mining<br/>
		Alexand(er|ra) Yst<br/>
		CS 4407: Data Mining and Machine Learning<br/>
		Unit 2
	</p>
</div>
<p>
	The assignment instructions for this essay aren&apos;t really very clear.
	They tell us to compare three examples from <strong>*one*</strong> of the following categories: databases, statistical packages, or $a[API]s.
	They then go on to tell us to compare our selected database, statistics package, and $a[API], as well as explain how they fit together within a single system.
	To do this, we&apos;d need to select one from each category, even though the instructions prior had told us to select three items from <strong>*one*</strong> category.
	It could be unclear wording in the first part, except that the database option clarifies and says we should compare a &quot;traditional&quot; database, an analytical database, <strong>*and*</strong> a non-relational database, which again points toward us choosing three items from one category.
	And to top it off, the hints as to what the grading rubric will be say to make sure to discuss all three categories.
	This lack of clarity as to what we&apos;re supposed to write about means we&apos;re not likely to meet the expectations for grading, because we can&apos;t even be sure what those expectations are.
	We&apos;ve got to just guess and hope for the best.
	I&apos;m guessing we&apos;re intended to write about one of each and explain how they can be used together in a single system, so that&apos;ll be my focus, but I&apos;ll also try to compare options just in case that&apos;s what was intended of us.
</p>
<p>
	There&apos;s also ambiguity as to what is meant by a &quot;traditional&quot; database.
	That&apos;s not really a database type.
	I&apos;m guessing what is meant is a <strong>*relational*</strong> database, not a &quot;traditional&quot; database.
	Again, given the lack of clarity, that&apos;s what I&apos;ll be going with.
</p>
<p>
	This paper is going to be a bit long.
	I&apos;m basically covering about double what I think was intended, just to make sure I cover the things I need to.
	If you don&apos;t like reading all this, blame the ambiguity of the assignment.
	I&apos;m honestly not sure which of this stuff is intended to be part of the assignment and which isn&apos;t.
	That said, I&apos;ll try to make this as brief as I can.
</p>
<h2>Databases</h2>
<p>
	This week, we studied non-relational databases, and compared them with relational databases.
	We didn&apos;t discuss statistical databases at all though.
	So let&apos;s start out by discussion relational and non-relational databases, then move on to discuss statistical databases.
</p>
<h3>Relational databases</h3>
<p>
	For many people, when they think of a database, they think of a relational database.
	Relational databases, such as MySQL, have a number of advantages.
	Their biggest advantage is that the software that manages them keeps these databases in a coherent state.
	Every potential change to the data - whether it be an insertion, a deletion, or an update - is checked to see if it would leave the database in a valid state.
	This means data can&apos;t be put where it doesn&apos;t belong, but also means that data must be kept consistent.
	Relational databases impose a rigid structure on the data they contain, and that structure may not be violated.
	Data on one table can also relate to data on another table.
	With the right table structures, you can avoid duplicating data by moving common data elements to their own table instead of including them in each and every row (Strauch, n.d.).
</p>
<p>
	These advantages all come at a computational cost though.
	Every potential change to the database must be gone over thoroughly to ensure it doesn&apos;t put the database into an invalid state, and these checks take processing power, and thus time.
	It can also be very difficult to store data that doesn&apos;t conform to a rigid structure when you&apos;re using a relational database.
	Relational databases don&apos;t function efficiently when spread out across multiple machines, either (Strauch, n.d.).
</p>
<h3>Non-relational databases</h3>
<p>
	Non-relational databases seek to eliminate the main problems associated with relational databases.
	Their main feature is that they don&apos;t impose rigid structure on the data.
	This removes all the integrity checks relational databases must perform, speeding up insertions, deletions, and updates.
	Without the rigid structure, it can also be much easier to store items that don&apos;t have as rigid a structure as a relational database prefers to store.
	Some non-relational database options provide greats support for splitting databases across multiple machines, too (Strauch, n.d.).
	Non-relational databases are said to scale better than relational databases because of this ability to easily be split cross multiple machines.
	Without being split up, you&apos;d instead need increasingly-capable single machines (Leavitt, 2010).
</p>
<p>
	Without rigid structure, tables can&apos;t relate to one another.
	Furthermore, without rigid integrity checks performed by the database software, data can become inconsistent.
	For example, a table listing all comments by a user may list a comment not listed in the table that lists all the comments for a particular thread, for a comment by that user on that thread.
	If such consistency is required, the checks must be performed outside the database software, and instead in the application (Strauch, n.d.).
</p>
<h3>Statistical databases</h3>
<p>
	Statistical database management systems are built to support the operations required or otherwise useful for statistical analysis.
	Common database management systems supply a few of the operations needed, but not nearly all of them.
	Statistical databases don&apos;t use $a[SQL], as it&apos;s not nearly flexible enough, so they use their own query language instead.
	Data is stored in a way that preserves semantics that would be lost when using relational databases.
	Many of the other benefits are those presented by relational databases as well, such as consistency constrains and database locking (Srivastava &apos; Ngo, n.d.).
</p>
<p>
	Two major disadvantages of statistical databases come to mind.
	The main one is that these databases are so tuned to deal with statistical data that they likely don&apos;t function well for other types of data.
	As we learned this week, specialised solutions are much better than generalised solutions for the use cases they are built for, but if your use case doesn&apos;t match, you&apos;ll need a different specialised solution instead.
	Secondly, the statistical operations they provide can be instead provided by a statistical operation package transparently to the user, meaning that users can have access to these operations with any type of database, not just a statistical database (Srivastava &apos; Ngo, n.d.).
</p>
<h2>Statistical packages</h2>
<p>
	Statistical packages provide us with the tools needed to perform statistical work with our data.
	In this course, the main one we&apos;ve really looked at is R.
</p>
<h3>S and S-Plus</h3>
<p>
	S was an old statistical package originally available to educational institutions, but once it was sold to Insightful, it eventually lost popularity.
	Insightful changed the way the product was marketed it, rebranding it as S-Plus in the process, and it wasn&apos;t freely-available to educational institutions any more.
</p>
<h3>LISP-STAT</h3>
<p>
	With the advent of LISP-STAT, S-Plus kind of fell out of use.
	LISP-STAT wasn&apos;t quite the same thing, but it was close enough and was much easier and less expensive to get ahold of.
	In fact, it was free software, which meant it could even be modified to fit the needs of its users.
	LISP-STAT was embedded in the Lisp interpreter, which meant that a full programming language was available to all that used it, making it highly flexible (Leeuw, n.d.).
</p>
<h3>R</h3>
<p>
	R is a free software statistical package designed to usurp S, an older proprietary statistical package.
	It pretty much replaced S.
	Users that liked the way S worked migrated from LISP-STAT to R, getting back to having the S environment without any of the baggage of the actual S-Plus software (Leeuw, n.d.).
	R can be run interactively for quick operations, but is fully capable of running S scripts, as well (R Core Team, 2018).
	That means R can be used in an automated fashion and in combination with other tools.
</p>
<h2>$a[API]s</h2>
<p>
	Our reading assignment didn&apos;t really discuss any specific statical packages, though it did drop a few names in the various tables presented, such as $a[Weka], Orange, and Hadoop.
	All three of these are free software, so you don&apos;t need to worry about malicious things hidden away in the code.
	We didn&apos;t really discuss $a[API]s this week though, so I&apos;m guessing it&apos;s not the details of the $a[API]s we&apos;re intended to discuss, but rather the benefits and drawbacks to each package.
	After all, learning three $a[API]s in one week so we can compare and contrast them is a bit excessive, and we&apos;d never be able to pull that off in such a limited time.
	I&apos;ll cover what these packages are very briefly;.
</p>
<h3>$a[Weka]</h3>
<p>
	$a[Weka] is intended as a collection of algorithms used for data mining (The University of Waikato, n.d.).
	$a[Weka] and hadoop serve very different purposes, and could be used together.
	Hadoop would take care of storage and distributed processing, while $a[Weka] would supply the algorithms used in said processing.
</p>
<h3>Orange</h3>
<p>
	Orange is used for data mining, but more interestingly, is used for data visualisation (University of Ljubljana, n.d.).
	It&apos;s also got a plug-in system that allows developers to extend the functionality of the product.
	Though I&apos;ve never had the chance to actually use the tool, it seems like the data visualisation would come in handy for understanding the patterns present in the data you&apos;re working with.
	As we learned last week, R offers the option to visualise data too, but this doesn&apos;t seem to be a strength $a[Weka] has.
	Again, Orange seems like a good tool to use in combination with Hadoop, but there&apos;s no need to combine it with $a[Weka], as Orange and $a[Weka] fill similar roles.
</p>
<h3>Hadoop</h3>
<p>
	Apache Hadoop is software used to set up distributed filesystems (The Apache Software Foundation, n.d.).
	It also facilitates the breaking down of data so it can be processed in a distributed manner as well.
	Unlike $a[Weka] and orange, Hadoop is able to store large quantities of data in a fault-tolerant way.
	If one server goes down, that same data can be found on other servers in the cluster (or in another data centre, as Hadoop can likewise operate across servers separated by vast distances).
</p>
<h2>A complete system</h2>
<p>
	Using one tool from each of these three categories, it&apos;s easy to see how a complete system could be used.
	For example, we could use Hadoop to store a database.
	This database could be a non-relational one, allowing all sorts of unstructured data to be kept without hassle.
	Hadoop would take care of both the distribution of the data across servers as well as the redundancy needed to avoid most data loss.
	(Some data loss is still likely due to the nature of non-relational databases, but that&apos;s data that failed to make it into the database, not data that was lost after having been successfully saved.)
	When we go to analyse our data, we could use one of the statistical packages, such as R or LISP-STAT, or one of the other $a[API]s, such as those provided by Orange or $a[Weka].
	If we wanted to visualise the data, Orange or R would probably be good choices.
</p>
<div class="APA_references">
	<h2>References:</h2>
	<p>
		The Apache Software Foundation. (n.d.). Apache Hadoop. Retrieved from <a href="https://hadoop.apache.org/"><code>https://hadoop.apache.org/</code></a>
	</p>
	<p>
		Leavitt, N. (2010, January 26). Will NoSQL Databases Live Up to Their Promise? Retrieved from <a href="http://leavcom.com/pdf/NoSQL.pdf"><code>http://leavcom.com/pdf/NoSQL.pdf</code></a>
	</p>
	<p>
		Leeuw, J. D. (n.d.). STATISTICAL SOFTWARE - OVERVIEW. Retrieved from <a href="http://gifi.stat.ucla.edu/janspubs/2009/reports/deleeuw_R_09a.pdf"><code>http://gifi.stat.ucla.edu/janspubs/2009/reports/deleeuw_R_09a.pdf</code></a>
	</p>
	<p>
		R Core Team. (2018, December 20). An Introduction to R: 14 OS facilities. Retrieved from <a href="https://cran.r-project.org/doc/manuals/R-intro.html#OS-facilities"><code>https://cran.r-project.org/doc/manuals/R-intro.html#OS-facilities</code></a>
	</p>
	<p>
		Srivastava, J., &amp; Ngo, H. J. (n.d.). Statistical Databases. Retrieved from <a href="https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.128.4820&amp;rep=rep1&amp;type=pdf"><code>https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.128.4820&amp;rep=rep1&amp;type=pdf</code></a>
	</p>
	<p>
		Strauch, C. (n.d.). NoSQL Databases. Retrieved from <a href="https://christof-strauch.de/nosqldbs.pdf"><code>https://christof-strauch.de/nosqldbs.pdf</code></a>
	</p>
	<p>
		University of Ljubljana. (n.d.). Orange - Data Mining Fruitful &amp; Fun. Retrieved from <a href="https://orange.biolab.si/"><code>https://orange.biolab.si/</code></a>
	</p>
	<p>
		The University of Waikato. (n.d.). Weka 3 - Data Mining with Open Source Machine Learning Software in Java. Retrieved from <a href="https://www.cs.waikato.ac.nz/ml/weka/"><code>https://www.cs.waikato.ac.nz/ml/weka/</code></a>
	</p>
</div>
END
);
